Skip to content

Keep activity claims through transient lease renewal failures; release an unrenewable claim as worker_lost (#352) - #353

Merged
rmcdaniel merged 4 commits into
mainfrom
general/352-activity-claim-renewal
Nov 20, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
general/352-activity-claim-renewal

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Fixes item 4 of #352. This is the Python counterpart of durable-workflow/sdk-php#445.

Bug

When an activity poll returns a task, ActivityLeaseAuthority.confirm_delivery sends one renew-lease request with a min(5 s, remaining) budget and no retry. A transport timeout, a connection reset, or a 5xx/429 answer raised out of poll_activity_task. The poll loop then discarded the claimed task (worker.py:4244-4277 on main). Nothing was completed, failed or released. The Server held the lease (300 s by default) and settled it as expired. With the default single attempt, that failed the workflow.

In steady state, renewals that failed until the local bound ran out also ended in a silent claim_aborted.

Change

Keeps working. A transient failure keeps the claim, at start or later. The activity runs under the lease it already holds, and renewal is retried inside that lease.

  • The backoff is the same in every SDK: the n-th retry waits uniform(0, min(4, 0.25·2^n)) seconds, with full jitter.
  • Every renewal exchange is bounded to leave a reserve of the lease (at most 10 s, at most a fifth of the grant).

Conservative lease clock. An unconfirmed grant is charged from the poll's start, because the Server stamps the lease only after it receives the poll. If that grant is already spent, nothing has run, so the unstarted claim is released directly. run() never starts an exhausted claim.

Stops before it releases (shared rule for all SDKs). Each claim's activity threads and callback processes are tracked on its lease authority. Each thread is tracked by a race-free guard wrapped around the activity: a thread that has not started yet is fenced so it never runs, and a started one counts as stopped only once it returns. If renewal has not succeeded when only the reserve is left, the execution is stopped:

  • an async activity is cancelled
  • a thread activity is asked to stop: its next heartbeat_sync() raises ActivityWorkerLost
  • a callback process is stopped

The worker then waits up to half of the reserve for every execution to actually stop. Only then does it release the claim, inside the remaining lease, with the cross-SDK wire contract:

  • POST /api/worker/activity-tasks/{id}/fail
  • failure.type = "worker_lost"
  • failure.class = "ActivityWorkerLost"
  • failure.non_retryable = false
  • a message naming the cause activity_lease_renewal_unavailable
  • failure.details = {"reason": "worker_lost", "cause": ...}

If the execution does not stop in time, for example a thread that never heartbeats or a coroutine that swallows cancellation, nothing is sent. The Server settles the attempt at lease expiry (lease_expired), so two executions of one attempt never overlap. A stale or late release is refused by the Server anyway (409 stale_attempt on 3.16.0).

Unchanged: a typed Server refusal (lease_expired, run_closed, other 4xx) still ends the claim as before. Fencing of late results is unchanged.

Rename. ActivityLeaseAuthorityLost becomes ActivityWorkerLost. The old name stays importable as a deprecated alias of the same class.

Workflow side. ActivityFailed.lease_loss_reason() returns:

  • "worker_lost" for a released claim
  • for a Server-settled ActivityLeaseExpiredException: the recorded failure_reason ("lease_expired" or "worker_deregistered"), defaulting to "lease_expired"
  • None for any other failure

The default retry policy is unchanged.

Docs. There is a new "Activity lease renewal" section in docs/sdk-reference.md, covering the backoff, the reserve, the stop-then-release rule, and lease_loss_reason().

Replay safety. Worker-side lease handling only. No workflow commands and no recorded history change.

Tests

tests/test_activity_lease_worker.py runs against a synthetic lease server with faults injected (timeout, reset, 503):

Case Expected result
Initial confirmation fails, then succeeds The activity completes, renewal is retried, no /fail is sent.
Mid-execution renewal is flaky, then recovers The activity outlives the original lease.
Renewal never recovers (async activity) Exactly one contract-conforming worker_lost /fail, inside the lease.
Async activity cancelled cooperatively Released.
Async activity swallows cancellation and outlives the stop window Nothing is sent.
Thread that stops at its next heartbeat_sync() Released, and only after the thread stopped.
Thread that never stops claim_aborted. The thread is still running and holding its slot, and nothing is ever sent, so the lease is left to expire.
Long poll followed by a failed confirmation The unconfirmed grant is charged from the poll start.
Unstarted claim whose conservative grant is spent Released, and the handler never runs.
Initial confirmation refused by the Server (lease_expired) Still raises.

tests/test_activity_lease_authority.py checks:

  • the backoff ceilings (0.25, 0.5, 1, 2, 4, 4 s), with the jitter spread over [0, ceiling]
  • that stop confirmation requires every thread and callback process to have stopped before the deadline
  • that the thread guard fences a thread that has not started

tests/test_activity_worker_lost.py checks the alias and lease_loss_reason().

Local results (Python 3.14, httpx 0.28.1):

  • pytest tests/ -m "not integration": 7807 passed, 3 skipped.
  • Lease worker tests: 29 passed, 3 consecutive runs.
  • ruff: clean.
  • mypy: clean.

🤖 Generated with Claude Code

A transport timeout, connection reset or 5xx/429 on the renew-lease
confirmation sent right after an activity poll raised
ActivityLeaseAuthorityLost out of the poll, and the worker discarded the
claimed task without completing, failing or releasing it. The Server
then held the lease until expiry and, with the default single attempt,
failed the workflow (same bug as durable-workflow/sdk-php#445).

Now the activity runs under the poll's grant while confirmation is
retried with backoff inside the lease. Every renewal is bounded to leave
a report reserve (at most 10 s, at most a fifth of the grant). If
renewal has still not succeeded when only that reserve is left, the
worker stops the execution and fails the attempt retryably through the
Server (ActivityLeaseAuthorityLost / activity_lease_renewal_unavailable)
instead of dropping it. Typed Server refusals still end the claim.

Worker-side lease handling only; no commands or history change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…#352)

Align the released-claim failure with the cross-SDK wire contract shared
with PHP and Rust: failure.type "worker_lost", failure.class
"ActivityWorkerLost", non_retryable false, the local cause
(activity_lease_renewal_unavailable) in the message, and details
{"reason": "worker_lost", "cause": <cause>}.

Rename ActivityLeaseAuthorityLost to ActivityWorkerLost, keeping the old
name as a deprecated alias of the same class so existing handlers keep
working.

Add ActivityFailed.lease_loss_reason(): "worker_lost" for a released
claim, the recorded failure_reason ("lease_expired" or
"worker_deregistered", default "lease_expired") for a Server-settled
ActivityLeaseExpiredException, otherwise None. ActivityFailed now keeps
the recorded failure_reason. The default retry policy is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel changed the title Keep activity claims through transient lease renewal failures; fail, never drop, an unrenewable claim (#352) Keep activity claims through transient lease renewal failures; release an unrenewable claim as worker_lost (#352) Nov 20, 2026
rmcdaniel and others added 2 commits November 20, 2026 04:07
Review follow-up, shared rule for all SDKs: a claim is released as
worker_lost only after its execution has confirmed stopped inside the
lease; otherwise nothing is sent and the Server settles lease_expired,
so two executions of one attempt never overlap.

- Thread futures and callback processes of a claim are tracked on its
  lease authority. On renewal exhaustion the execution is stopped (async
  cancelled, thread asked to stop via its next heartbeat_sync, callback
  stopped) and awaited for up to half of the release reserve. Only then
  is the release sent, within the remaining lease.
- An unconfirmed grant is charged from the poll's start, the
  conservative bound. If it is already spent, the unstarted claim is
  released directly, and run() never starts an exhausted claim.
- Renewal retries use the cross-SDK backoff: uniform(0, min(4,
  0.25 * 2**n)) seconds (full jitter). Documented in the SDK reference.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…352)

Wrap a claim's synchronous activity before it is handed to the thread
pool. A thread that has not started yet is fenced when the release asks,
so it never runs afterwards; a started one counts as stopped only once
its function has returned or raised. This also keeps the shared
thread-pool code path (a replay-guarded symbol) unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit cd5cff4 into main Nov 20, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant